A Multi-Modal Oculography-Based Mouse Controlling System: Via Facial Expressions & Eye Movement

Hand free mouse controlling systems is an effective and essential research area in human computer interaction (HCI), as they can be an ideal communication and interaction means for the physical disabled persons, who don’t have means for communication except their moving eyes. In addition, these types of application provide usability and welfare for the normal users in the computer interaction. This paper presents a fully automatic multi-modal oculography-based mouse controlling system. This is done via integration between the recognized facial expressions with the eye movement and utilizing them in the mouse controlling. This paper presents a powerful multi-detector technique to localize the key facial feature points so that contours of the facial components are sampled such as the eyes, nostrils, chin, and the mouth. Also, the beholder’s gaze is estimated. Based on the 66 extracted facial features points, 20 geometric formulas (GF) and 15 ratios (Rs) are calculated, and the classifier based on rule-based reasoning approach are then formed for both of the gaze direction and the facial expression (Normal, Smiling or Surprising). Tests were performed for 7 users with different ages and different facial characteristics then their overall satisfaction was measured. Our proposed multi-modal system obtained about 91.1% to 94.6% user satisfaction.